{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorGraph Mechanics\n",
    "In this IPython notebook, we will cover more advanced aspects of the `TensorGraph` framework. In particular, we will demonstrate how to share weights between layers and show how to use `DataBag` to reduce the amount of overhead needed to train complex `TensorGraph` models.\n",
    "\n",
    "Let's start by defining a `TensorGraph` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Warning: No xgboost installed on your system\n",
      "Attempting to run xgboost will throw runtime errors\n",
      "Warning: No pyGPGO.covfunc installed on your system\n",
      "Attempting to run pyGPGO.covfunc will throw runtime errors\n",
      "Warning: No pyGPGO.acquisition installed on your system\n",
      "Attempting to run pyGPGO.acquisition will throw runtime errors\n",
      "Warning: No pyGPGO.surrogates.GaussianProcess installed on your system\n",
      "Attempting to run pyGPGO.surrogates.GaussianProcess will throw runtime errors\n",
      "Warning: No pyGPGO.GPGO installed on your system\n",
      "Attempting to run pyGPGO.GPGO will throw runtime errors\n"
     ]
    }
   ],
   "source": [
    "import deepchem as dc\n",
    "from deepchem.models.tensorgraph.tensor_graph import TensorGraph\n",
    "\n",
    "tg = TensorGraph(use_queue=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to construct an architecture that has two identical feature inputs. Let's call these feature inputs `left_features` and `right_features`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from deepchem.models.tensorgraph.layers import Feature\n",
    "\n",
    "left_features = Feature(shape=(None, 75))\n",
    "right_features = Feature(shape=(None, 75))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now apply a nonlinear transformation to both `left_features` and `right_features`. We can use the `Dense` layer to do so. In addition, let's make sure that we apply the same nonlinear transformation to both `left_features` and `right_features`. To this, we can use the `Layer.shared()`. We use this method by initializing a first `Dense` layer, and then calling the `Layer.shared()` method to make a copy of that layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deepchem.models.tensorgraph.layers import Dense\n",
    "\n",
    "\n",
    "dense_left = Dense(out_channels=1, in_layers=[left_features])\n",
    "dense_right = dense_left.shared(in_layers=[right_features])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now combine these two transformed feature layers by addition. We will assume this network is being used to solve a regression problem, so we will introduce a `Label` that stores the true regression values. We can then define the objective function of the network via the `L2Loss` between the added output and the true label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deepchem.models.tensorgraph.layers import Add\n",
    "from deepchem.models.tensorgraph.layers import Label\n",
    "from deepchem.models.tensorgraph.layers import L2Loss\n",
    "from deepchem.models.tensorgraph.layers import ReduceMean\n",
    "\n",
    "output = Add(in_layers=[dense_left, dense_right])\n",
    "tg.add_output(output)\n",
    "\n",
    "labels = Label(shape=(None, 1))\n",
    "batch_loss = L2Loss(in_layers=[labels, output])\n",
    "# Need to reduce over the loss\n",
    "loss = ReduceMean(in_layers=batch_loss)\n",
    "tg.set_loss(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now randomly sample an artificial dataset we can use to train this architecture. We will need to sample the `left_features`, `right_features`, and `labels` in order to be able to train the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import numpy.random\n",
    "\n",
    "n_samples = 100\n",
    "sampled_left_features = np.random.rand(100, 75)\n",
    "sampled_right_features = np.random.rand(100, 75)\n",
    "sampled_labels = np.random.rand(75, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How can we train `TensorGraph` networks with multiple `Feature` inputs? One option is to manually construct a python generator that provides inputs. The tutorial notebook on graph convolutions does this explicitly. For simpler cases, we can use the convenience object `DataBag` which makes it easier to construct generators. A `DataBag` holds multiple datasets (added via `DataBag.add_dataset`). The method `DataBag.iterbatches()` will construct a generator that peels off batches of the desired size from each dataset and return a dictionary mapping inputs (`Feature`, `Label`, and `Weight` objects) to data for that minibatch. Let's see `DataBag` in action.\n",
    "\n",
    "Note that we will need to wrap our sampled Numpy arrays with `NumpyDataset` objects for our call to work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from deepchem.data.datasets import Databag\n",
    "from deepchem.data.datasets import NumpyDataset\n",
    "\n",
    "databag = Databag()\n",
    "databag.add_dataset(left_features, NumpyDataset(sampled_left_features))\n",
    "databag.add_dataset(right_features, NumpyDataset(sampled_right_features))\n",
    "databag.add_dataset(labels, NumpyDataset(sampled_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now train this architecture! We need to use the method `TensorGraph.fit_generator()` passing in a generator created by `databag.iterbatches()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ending global_step 200: Average loss 0.472205\n",
      "TIMING: model fitting took 0.273 s\n"
     ]
    }
   ],
   "source": [
    "tg.fit_generator(\n",
    "    databag.iterbatches(epochs=100, batch_size=50, pad_batches=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should now be able to construct more sophisticated `TensorGraph` architectures with relative ease!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
